Color night vision cameras, systems, and methods thereof

ABSTRACT

Disclosed are improved methods, systems and devices for color night vision that reduce the number of intensifiers and/or decrease noise. In some embodiments, color night vision is provided in system in which multiple spectral bands are maintained, filtered separately, and then recombined in a unique three-lens-filtering setup. An illustrative four-camera night vision system is unique in that its first three cameras separately filter different bands using a subtractive Cyan, Magenta and Yellow (CMY) color filtering-process, while its fourth camera is used to sense either additional IR illuminators or a luminance channel to increase brightness. In some embodiments, the color night vision is implemented to distinguish details of an image in low light. The unique application of the three-lens subtractive CMY filtering allows for better photon scavenging and preservation of important color information.

CROSS REFERENCE TO RELATED APPLICATIONS

This Application is a continuation of U.S. application Ser. No.16/517,451, filed 19 Jul. 2019, which was issued as U.S. Pat. No.10,582,173 on 3 Mar. 2020, which is a Continuation of U.S. applicationSer. No. 15/877,300, filed 22 Jan. 2018, which was issued as U.S. Pat.No. 10,362,285 on 23 Jul. 2019, which is a Continuation of U.S.application Ser. No. 15/357,788, filed 21 Nov. 2016, which was issued asU.S. Pat. No. 9,894,337 on 13 Feb. 2018, which is a Continuation of U.S.application Ser. No. 14/730,172, filed 3 Jun. 2015, which was issued asU.S. Pat. No. 9,503,623 on 22 Nov. 2016, which claims priority to U.S.Provisional Application No. 62/007,317, filed 3 Jun. 2014, wherein eachis incorporated herein in its entirety by this reference thereto.

FIELD OF THE INVENTION

At least one embodiment of the present invention pertains to color nightvision cameras and imagers, and associated methods, including thereduction of intensifiers to save cost and weight, as well as advancedprocessing techniques.

BACKGROUND

Monocular and stereo monochrome night vision systems are in widespreaduse in law enforcement, military, and recreational applications. Thesetraditional systems are sensitive to the visible and infrared spectrum,but during light gathering and photon scavenging within the microchannelplate (MCP) photomultiplier tube, all color information is lost, whichleaves a monochrome image illuminated by the green P43 phosphor at therear of the tube. The application of the green P43 phosphor is chosenprimarily because of the human eye's sensitivity toward this wavelengthdue to peak gain. As well, green wavelengths are the least bright signalviewable by the human eye, which, in turn, creates energy efficienciesin the system while concurrently keeping the viewer night adapted.

This image composed of monochromatic shades of green, leaves viewersstarved for color information. Since color perception is a key factor insituational awareness, a full-color night vision system is useful fordistinguishing objects from a background and identifying them.

A number of techniques for color night vision have been proposed andimplemented, but generally require complex and expensive mechanisms andimagers. One technique is to require three input/output channels torepresent the color spectrum, typically RGB (red, green, and blue),increasing to an impractical six intensifiers for stereo color vision.

An alternative technique is to use a spinning wheel or other mechanicalapparatus to rapidly change color filters at the entrance and exit of asingle intensifier. Both techniques drive up the cost and complexity,with increased weight and bulk that is especially critical for head-worngoggles.

The additional complexity associated with these approaches has slowedthe adoption of color night vision systems. Thus, while it would bepossible to simply combine the existing stereo, monochrome techniquesand monocular, color techniques to yield stereo, color night visionfunctionality, the resulting system would be so complex and cumbersomeas to make field usage impractical.

Another major issue with all night vision systems is image noise. Allimage amplification tubes used in night vision devices have inherentnoise, seen as a distinctive speckling or scintillation of individualpixels from frame to frame, becoming more prominent as the gain(amplification) and/or operating temperature is increased. In alow-light environment, the shot noise from random photon arrivals tendsto dominate the image and preclude its interpretation. Small amounts ofcolor channel noise can easily dominate the chrominance signal inreal-world scenes.

Ideally, a color night vision system should adapt to changing luminanceand scenic parameters such that it automatically presents an easilyinterpreted, accurate rendering of the scene. A practical system must beportable, rugged, and not overly costly.

BRIEF DESCRIPTION OF THE DRAWINGS

One or more embodiments of the present invention are illustrated by wayof example and not limitation in the figures of the accompanyingdrawings, in which like references indicate similar elements. The patentor application file contains at least one drawing executed in color.Copies of this patent or patent application publication with colordrawing(s) will be provided by the Office upon request and payment ofthe necessary fee.

FIG. 1 is a front view of an illustrative embodiment of a color nightvision camera.

FIG. 2 is a right side view of an illustrative embodiment of a colornight vision camera.

FIG. 3 is a left side view of an illustrative embodiment of a colornight vision camera.

FIG. 4 is a top view of an illustrative embodiment of a color nightvision camera.

FIG. 5 is a bottom view of an illustrative embodiment of a color nightvision camera.

FIG. 6 shows four identical folded-optic camera assemblies arrayedaround the central combining mirror assembly.

FIG. 7 is an illustrative chart of absolute emission spectra of commonphosphors, per Watt absolute emission spectra of excitation power.

FIG. 8 depicts a live-action image taken from a typical night visioncamera.

FIG. 9 depicts a live-action image taken with a color night visioncamera, in the same dark conditions as shown in FIG. 8.

FIG. 10 is a schematic diagram of an illustrative color night visionsystem using subtractive CMY color filters.

FIG. 11 is a detailed schematic diagram of a first portion of anillustrative color night vision system using subtractive CMY colorfilters.

FIG. 12 is a detailed schematic diagram of a second portion of anillustrative color night vision system using subtractive CMY colorfilters.

FIG. 13 is a chart that shows quantum efficiency as a function ofwavelength for illustrative CMY color curves, as compared to that of RGBcolor curves.

FIG. 14 shows an illustrative schematic diagram of an embodiment of acolor night vision system which includes four intensifier tubeassemblies, four CCD imagers, and a pair of output displays.

FIG. 15 shows an illustrative schematic diagram of an embodiment of anillustrative color night vision system that includes four intensifiertube assemblies, and beamsplitters between each of the intensifiers andeye viewing.

FIG. 16 shows an illustrative schematic diagram of an embodiment of acolor night vision system that includes three intensifier tubeassemblies to provide a luminance channel and a chrominance channel.

FIG. 17 shows an illustrative schematic diagram of an embodiment of acolor night vision system that includes two intensifier tube assemblies810.

FIG. 18 shows an illustrative schematic diagram of an alternateembodiment of a color night vision system that includes two intensifiertube assemblies, which characterizes the color input by each fiber of anintensifier, enabling the system to reproduce a color image using onlyone intensifier per eye.

FIG. 19 is a process flowchart of an illustrative method for reducingimage noise in a color night vision device, by comparing output of thecolor channels.

FIG. 20 is a process flowchart of an illustrative method for reducingimage noise in a color night vision device, using a plurality ofbuffers.

FIG. 21 is a process flowchart of an illustrative method for reducingimage noise in a color night vision device, which applies a non-localimage denoising algorithm to reduce the noise.

FIG. 22 is a schematic view of a circular filter disk for a color nightvision camera, wherein the filter disk is in a first position, and canbe rotated in an image path, to vary the effectiveness of the colorfilter.

FIG. 23 is a schematic view a circular filter disk for a color nightvision camera, wherein the filter disk is in a second position, and hasbeen rotated to a second position, to vary the effectiveness of thecolor filter.

FIG. 24 is a simplified schematic diagram of an illustrative embodimentof a system for CMY-filtered sequential color night vision.

FIG. 25 is a detailed view of light transmitted through an inclinedfilter.

FIG. 26 is a flowchart of an illustrative method for CMY-filteredsequential color night vision.

FIG. 27 is a simplified schematic diagram of motion compensation fortemporal scanning color night vision.

FIG. 28 is a flowchart of an illustrative method for motion compensationfor temporal scanning color night vision.

DETAILED DESCRIPTION

References in this description to “an embodiment”, “one embodiment”, orthe like, mean that the particular feature, function, structure orcharacteristic being described is included in at least one embodiment ofthe present invention. Occurrences of such phrases in this specificationdo not necessarily all refer to the same embodiment. On the other hand,the embodiments referred to also are not necessarily mutually exclusive.

Introduced here are improved methods, systems and devices for colornight vision that reduce the number of intensifiers and/or decreasenoise.

In certain embodiments, the technique introduced here involves aneffective color night vision system in which multiple spectral bands aremaintained, filtered separately, and then recombined in a uniquethree-lens-filtering setup. This four-camera night vision system isunique in that its first three cameras separately filter different bandsusing a subtractive Cyan, Magenta and Yellow (CMY) colorfiltering-process, while its fourth camera is used to sense eitheradditional IR illuminators or a luminance channel to increasebrightness.

In some embodiments, an improved method and implementation of colornight vision is implemented to distinguish details of an image in lowlight. The unique application of the three-lens subtractive CMYfiltering allows for better photon scavenging and preservation ofimportant color information.

CMY-Filtering Color Night Vision Camera

The following describes an effective color night vision system in whichmultiple spectral bands are maintained, filtered separately, and thenrecombined in a unique three-lens-filtering setup.

FIG. 1 is a front view of an illustrative embodiment of a color nightvision camera 10. FIG. 2 is a right side view 100 of an illustrativeembodiment of a color night vision camera 10. FIG. 3 is a left side view160 of an illustrative embodiment of a color night vision camera 10.FIG. 4 is a top view 200 of an illustrative embodiment of a color nightvision camera 10. FIG. 5 is a bottom view 240 of an illustrativeembodiment of a color night vision camera 10.

The illustrative color night vision camera 10 seen FIG. 1 includes acolor night vision camera body 12 that is attached 34, e.g., 34 a, 34 b,to a camera housing 14. The illustrative color night vision camera body12 shown in FIGS. 1-5 has a width 18 (FIG. 1), e.g., 16.50 inches, aheight 102 (FIG. 2), e.g., 18.27 inches, a body depth 162 (FIG. 3),e.g., 12.51 inches, and an overall depth 162 (FIG. 3), e.g., 15.75inches. The illustrative camera housing 14 seen in FIGS. 1-5 has a width22 (FIG. 1), e.g., 26.03 inches.

FIG. 6 is partial cutaway view 300 of four identical folded-optic cameraassemblies 302, e.g., 302 a-302 d arrayed around the central combiningmirror assembly 304 for an illustrative embodiment of a color nightvision camera 10. Each assembly 302, e.g., 302 a, is comprised of ahigh-resolution monochrome camera-link imager 306, a relay optic 308, animage-intensifier filter 310 iris motors 312, and an imaging lens 314.

This four-camera night vision system is unique in that its first threecameras 302, e.g., 302 a-302 c separately filter different bands using asubtractive Cyan, Magenta and Yellow (CMY) color filtering process,while its fourth camera 302, e.g., 302 d, is used to sense eitheradditional IR illuminators, or a luminance channel, e.g., 1304 (FIG.16), to increase brightness. This subtractive CMY filtering processdiffers from the typical additive Red, Green and Blue (RGB) filters.

FIG. 7 is an illustrative chart 400 of absolute emission spectra 404 ofcommon phosphors, including P11 phosphor 406, P20 phosphor 408, P43phosphor 410, P46 phosphor 412, and P47 phosphor 414. The energyconversion 404 is shown per Watt absolute emission spectra of excitationpower, as a function of wavelength 402. As seen in FIG. 7, P43 phosphor410 is strongly peaked in a narrow band centered in the green at 545 nm.

FIG. 8 depicts a live-action image 500 taken from a typical night visioncamera. The image is in monochromatic shades of green, due to the P43phosphor 410 that is commonly used in such cameras. The heavyintensifier speckling adds to the noise and difficulty of discerningobjects within the scene.

FIG. 9 depicts a live-action image 600 taken with a color night visioncamera 10. This picture 600 was taken with no active illuminatinginfrared (IR) device, in the same dark conditions as shown in FIG. 8.

CMY filtering effectively allows the camera 10 and associated system 300to capture more color information, in part, because each filter capturestwo sets of color information. For example, cyan filtering capturesgreen and blue wavelengths, magenta filtering captures red and bluewavelengths, and yellow filtering captures red and green wavelengths.

By doubling the ability to capture photons, the camera 10 and associatedsystem 700 increases the camera's aperture, without increasing thephysical device's size. This high-quality hardware solution can also becoupled with an integrated software package and associated processingsystem.

Three-Lens CMY Filtering

FIG. 10 is a schematic diagram of an illustrative color night visionsystem 700 using subtractive CMY color filters 812 (FIG. 11). FIG. 11 isa detailed schematic diagram 800 of a first portion of an illustrativecolor night vision system 700 using subtractive CMY color filters 812.FIG. 12 is a detailed schematic diagram 900 of a second portion of anillustrative color night vision system using subtractive CMY colorfilters. FIG. 13 is a chart 1000 that shows quantum efficiency 1002 as afunction of wavelength 402 for illustrative CMY color curves 1004 c, 104m, 1004 y, as compared to that of RGB color curves 1004 r, 1004 g, 1004b.

The unique application of the three-lens subtractive CMY filteringallows for better photon scavenging and preservation of important colorinformation. CMY consists of Cyan, Magenta and Yellow, which are thechromatic complements of the additive primaries Red, Green and Blue(RGB), respectively. However, CMY are subtractive primaries, due totheir effect in subtracting color from white light. Cyan is acombination of Blue plus Green, Magenta is a combination of Red plusBlue, and Yellow is a combination of Red plus Green. The system 700,night vision camera 10, and associated methods disclosed herein cantherefore be configured to effectively double the amount of photonscaptured, thereby increasing our aperture without adding bulk.

A lookup table can be used to gamma-correct the CMY data, prior toconverting it to the RGB color space, to be displayed on a displayscreen 760 (FIG. 12), e.g., an LCD display screen 760. Since CMY is madeup of RGB components, the following linear equations (masking equations)can be used to convert between CMY and RGB space:

C=1−R; such that R=1−C;

M=1−G; such that G=1−M; and

Y=1−B; such that B=1−Y.

As well, this unique arrangement allows the system 700 to provide ahigher quality image when removing noise, for embodiments wherein allcolor information is spatially encoded and separated on multipleintensifier tubes 810 (FIG. 11).

From the standpoint of efficiency, this CMY subtractive color system ishighly advantageous, but there are more advantages as well. Because twosubtractive color channels 710, e.g., 710 a and 710 b, have overlappinginformation, a bright “pop” of image noise originating in one of theintensifier tubes will not be seen in its neighbor, and therefore can beunderstood to be a noise event, and removed from the image 712.

In general, an increase in photon capture means a decrease in noise. Aswell, a parallel light path using a relay system increases quantumefficiencies due to the light path. The separation of these light pathsis 22 mm or less from being coaxial. This is not ideal, but with thelight paths being so close, the end result is less than 1 pixeldisparity at distances of 40 feet or more.

Spatial and Temporal Integration

The sensitivity of the system 700 can be increased by integrating one ormore color channels 710, e.g., between a red channel 710 r, a greenchannel 710 g, and a blue channel 710 b, either spatially by mergingpixels, or temporally by combining light gathered over several frames.This increases the overall brightness of the available image, at theexpense of reduced resolution from pixel merging or smearing of movingobjects.

When merging images over several frames, an exponential filter can beused:

Alpha(1-alpha) filter:

Filter value=alpha*measurement+(1-alpha)*previous value;

wherein alpha is scaled to the amount of movement in the image, on apixel by pixel basis, as determined by differential temporal analysis.

Image Processing

In some current embodiments, the system includes a graphics processingunit (GPU) 740, which can provide features such as any of CCD noiseremoval 902, a hybrid low pass filter 904, multi-channel (CMY) imageintensifier despeckle 906, color conversion 908, Gaussian differentialmotion analysis 910, temporal integration 912, motion integration 914,digital zoom 916, and automatic brightness leveling 918.

In some GPU embodiments 740, the image processing can be pre-formedusing the superior processing available in current graphics cards, whichin turn allows for high-resolution motion video, with minimal lag. Thelast few years have seen tremendous growth in graphics card capability,with graphics card performance in top of the line cards now more thandouble that currently implemented in night vision cameras.

Other key features can include intelligent time integration, whichallows for selective image sharpness and brightness enhancement. Normalnight vision systems are subject to the response time of the human eyeand the way the eye reacts to the phosphors used in night vision scopes.Intelligent integration provides integration of areas where there is nosensed motion, to produce an image where typical night vision wouldfalter. As well, this decreases the image lag typically associated withsuch integration techniques. While the existing camera-augmented nightvision systems have fixed-pattern noise and fixed-column noisereduction, our added line-noise reduction filters out nearly all of theremaining noise. This leaves no negative effect on an integrated imageand improves the quality of the live image.

Advanced noise removal techniques are implemented to further clarify theimage quality. High frequency pattern noise, dark current, andhigh-pass-blur filters are all implemented on the graphics card.

Software improvements that aid in increasing system gain and noisereduction include:

-   -   Identifying all sources of noise from intensifiers and all other        hardware in the imaging pipeline, then developing filters that        are specific to each source.    -   Developing a per-pixel integration factor with motion detection,        which gives us a brighter, clearer, more accurate image while        being able to detect and present motion information.    -   Despeckling the CMY color filter. Due to the use of CMY color        channels as source (and due to overlap of CMY in RGB color        space) image intensifier noise is detected that shows up as        errantly bright speckles in the live and integrated images.        Noise reduction is then achieved by comparing channels and        scoring or voting which channel has intensifier noise. This        method is not limited to CMY color channels, and can be extended        into multiple channels.    -   Developing a low-pass blur filter. The system maintains the        integrity of low-pass data and blurs high-pass data, which in        turn reduces shot noise while improving image detail over the        original blur.    -   Developing a motion outline Sobel edge-detection pass, which in        turn would highlight motion edges with a red color line. In        other words, it would effectively draw a red line around moving        images as an alert mechanism.    -   Development of customary camera controls. These controls can        include:        -   Light—In bright lighting conditions, it closes the iris            aperture; in dark lighting, it controls the brightness and            contrast; in all lighting conditions, it controls color            correction, gamma, saturation, and motion detection            multipliers.        -   Motion—In high-motion scenes, integration is reduced and            motion detection is increased; in static scenes integration            is increased and the image clarity is improved.        -   Color—The image can be represented in full color (RGB); the            image can be represented in full infrared (IR) as gray; the            image can be represented in full color plus IR as red;            lastly the image can be represented in full IR as red only.

Due to low-light gathering capabilities, night vision systems tend tohave a lot of shot noise associated with them. On the quantum level,photon shot noise stems from the randomness associated with photonarrivals. Random photon arrivals typically follow a Poissondistribution, whose signal-to-noise ratio increases as the square-rootof the light intensity. At very low light levels, where only a fewphotons are captured per pixel per frame, temporally fluctuating shotnoise can dominate the image and hinder interpretation.

Additionally, image intensifiers produce dark noise, which mimics theappearance of random photon arrivals, but is not correlated with theimage. It characteristically will produce image speckles, which areindependent within each intensifier and therefore are amenable toremoval through the detection of uncorrelated image speckles. Since darknoise sets a fundamental lower bound to the light levels, which can bereliably detected, it is greatly advantageous to minimize dark noiseeffects through intelligent filtering techniques. Dark noise filtrationas well as high frequency pattern noise removal, dark currentsubtraction, and low-pass blur filters, are all implemented in real timeon the high-performance graphics card.

A simplified color night vision system can be achieved by reducing thenumber of intensifiers 810 from the conventional six for stereo vision(three per eye) to four, three, or two, e.g., one for monocular vision.

FIG. 14 shows an illustrative schematic diagram 1100 of an embodiment ofa color night vision system 700 b which includes four intensifier tubeassemblies 810, four charge coupled device (CCD) imagers 820, and a pairof output displays 760, e.g., 760 a, 760 b. The optical axes of the fourintensifier tube assemblies 810 are aligned parallel to one another,with two intensifier tube assemblies 810 offset from one another alongone (preferably horizontal) dimension and the two other intensifier tubeassemblies 810 offset from one another along a perpendicular (preferablyvertical) dimension. The segments connecting the offset pairs ofintensifier tube assemblies 810 preferably intersect one another attheir respective midpoints.

Each intensifier tube assembly 810 includes a filter 812, a front lens814, a light intensifier 816, and a phosphorous screen 818, which actsas an image relay. A CCD imager 820 associated with each intensifiertube assembly 810 digitizes the output of the light intensifier 810, asrendered on the phosphorous screen 818. The outputs of the four CCDimagers 820 are then each assigned to a color channel 710. In theillustrative system 700 b seen in FIG. 14, one output 1102 b is assignedto a red channel 710 r, one output 1102 c is assigned to a red channel710 r, and two outputs 11021, 1102 d are assigned to green channels 710g. Preferably, the vertically offset intensifier tube assemblies 810 areassigned to red and blue channels 710 r and 710 b and the horizontallyoffset intensifier tube assemblies 810 are assigned to green channels710 g, termed the “right” and “left” channels, respectively.

The color channels 710 are then combined to form two composite colorimages displayed on the pair of output displays 760, e.g., 760 a, 760 b.So that the colors within the composite color images closely correspondto the actual color of the viewed subject matter, the filter 812 withineach intensifier tube assembly 810 is selected to pass wavelengths oflight corresponding to the assigned color channel 710. For example, thefilter 812 within the intensifier tube assembly 810 assigned to the redchannel 710 seen in FIG. 14 can be a relatively broad, notch-pass filterpassing red wavelengths of light.

Each composite color image is composed from one of the two greenchannels, and a shifted representation of each of the red and bluechannels. To create the shifted representation, the pixels within theimagery received from the channel 710 are first displaced, to compensatefor the (preferably vertical) offset between the red channel 710 r andthe blue channel 710 b. The pixels are then displaced along theperpendicular (preferably horizontal) dimension in one direction, forthe left composite color image, and in the opposite direction for theright composite image.

The human visual system is capable of relying upon depth informationpresent within one spectral band, even if the visual information incomplementary bands does not contain similar depth information. Forexample, U.S. Pat. No. 6,597,807 notes that “[i]f the fusion of one ofthe colored imagery layers produces depth perception information in oneportion of the scene whereas the other two colored layers do not, thehuman mind can retain the information from this color and discard theinputs from the other two colors as noise”.

Thus, when viewed separately by the left and right eyes of the user, thecomposite color images provide an effective stereo imagery pair, despitean absence of stereo information in the red and blue color channels 710r, 710 b. Furthermore, the color night vision system 700 b can maximizethe effectiveness of the stereo pair, by presenting the stereoinformation in the green channel 710 g displaying wavelengths to whichthe human eye is most sensitive. The color night vision camera 10 andsystem 700 b can thus reduce the number of intensifier tube assemblies810 to four from six (as would be required by atwo-intensifiers-per-RGB-channel device), with little reduction instereo imaging quality. The resulting color night vision camera 10 cantherefore be comparatively light and compact, and therefore moresuitable for use in head-mounted applications.

Generally, the magnitude of the displacement applied to the pixels ineach dimension is dependent on the offset between the intensifier tubeassemblies 810 along that dimension and the distance at which thesubject matter is located, with subject matter at lesser distancesrequiring a greater displacement. In one variation of the color nightvision system 700, a “typical subject matter distance” is determined forthe system at the time of manufacture (presumably based on the intendedusage of the device). All pixels within the image are displaced inaccordance with this distance at all times. Those portions of the imagecontaining subject matter at a distance substantially different than thetypical distance will exhibit blue and red (or perhaps purple) fringing.To minimize this effect, the (preferably vertical) offset between redand blue intensifier tube assemblies 810 can be reduced as far asallowed by the physical diameters of the intensifier tube assemblies810.

In another variation of the color night vision system 700, the distanceupon which the pixel displacement is based can be adjusted by the user.In this variation, the displacement remains constant for all pixelswithin an image, but can vary in time. Alternatively, a time varyingdisplacement can be automatically determined based on an “averagesubject matter distance” computed from a stereo correspondence performedupon the two green channel images and applied to all pixels within theimage. In yet another variation of the color night vision system 700,the pixel displacement varies both in time and within an image (frompixel to pixel) based on a dense correspondence computed in real-timefor the two green channel images.

As the blue and red fringing noted above can prove distracting in somesituations, the color night vision system 700 preferably provides aswitch or button with which the user can disable the color compositionfunctionality. When the color composition is disabled, the compositeimages are composed directly from the green channels to provide aconventional stereo, monochrome image pair.

Variations of the color night vision system 700 based on color spacesother than RGB are also possible. In one such variation, based on theCMY color space, stereo cyan channels are combined with shiftedrepresentations of monocular magenta and yellow channels 710 to createthe composite color channel.

FIG. 15 shows an illustrative schematic diagram of an embodiment of anillustrative color night vision system 700 c which includes fourintensifier tube assemblies 810 and beamsplitters 1148 between each ofthe intensifiers 816 and eye viewing 1150, e.g., 1150 a, 1150 b.

The green channel 710 g can be used for luminance information to extractthe maximum quality image possible. The color night vision system 700 cseparates the green channel for direct stereo viewing, which containsthe brightest and sharpest image information, from other color channels710, which are viewed in monocular vision, thus reducing the number onintensifiers 810 required from six to three or four.

The illustrative color night vision system 700 c seen in FIG. 15 hasfour image intensifier tube assemblies 810. Two of the intensifierassemblies 810 are fitted with a green pass filter at the front of thetube and placed for direct viewing by the left and right eyes. Thisprovides a conventional monochrome stereo image to the user at the greenfrequencies to which the human eye is most sensitive. A beamsplitter1148 is interspersed between each intensifier and the eye.

The vast majority of image detail lies in the luminance channel, withthe chrominance channel adding very little information. Since human eyesare most sensitive to the green bandwidth, it can be advantageous tomake this channel supply the luminance information without obstructionby color filters.

To provide color information, two additional image intensifiers 810 arevertically stacked between the green-filtered direct view intensifiers810, one with a red filter 812 and one with a blue filter 812. The axesof all four intensifiers 810 are aligned. Monochrome cameras 1144 viewthe red- and blue-filtered intensifiers 816, and send their colorchannels to an organic light-emitting diode (OLED) display 1146 besideeach beamsplitter 1148. The beamsplitter 1148 combines the direct viewgreen channel with the displayed red and blue channels.

To avoid false stereo cues, the red and blue channels 710 can beslightly defocused. An alternative method to avoid false stereo cues isto use a beamsplitter 1142 on the input side as well, with part of thelight going to the direct view stereo green channel and some going tothe separate red and blue mono channels 710.

Optionally, a CCD camera 820 (FIG. 14) on the opposite side of thebeamsplitter 1148 can view the green channel 710 g, if furtherprocessing and independent display are desired.

In other embodiments, the red, green, and blue channels 710 can bereplaced with magenta, cyan, and yellow channels, or the red and blueintensifiers 816 and cameras, e.g., 1144, can be replaced with IR and UVsensitive cameras, to permit overlay of broad spectrum visualinformation.

The result is color night vision using only four intensifiers 810, withdirect viewing 1150 a, 1150 b of the most important channel (green) 710g, in terms of brightness and sharpness.

FIG. 16 shows an illustrative schematic diagram of an embodiment of acolor night vision system 700 d that includes three intensifier tubeassemblies 810, to provide a luminance channel 1190 and a chrominancechannel 1192. The illustrative color night vision system 700 d seen inFIG. 16 takes advantage of the fact that the vast majority of imagedetail lies in the luminance channel 1190, while a chrominance channel1192 (FIG. 16) adds very little information. Since human eyes are mostsensitive to the green bandwidth, the illustrative color night visionsystem 700 d see in FIG. 16 makes this channel 710 supply the luminanceinformation, without obstruction by color filters. This separation intochrominance 1192 and luminance 1190 channels results in high spatial andtemporal resolution, and potentially further reduces the number ofintensifiers 810 to a total of three.

-   -   A beam splitter 1182 divides light coming in from an imaging        lens into two light paths 706, e.g., 706 a, for green, which is        treated as luminance 1190, and a second band 706, with all other        frequencies, which is treated as chrominance 1192.    -   The green, or luminance, band 706 a is fed to an image        intensifier 816 for direct viewing 760, while the other        (chrominance) channels 706 b are sent to the separate red and        blue channels 710 r and 710 b, as above. Alternatively, a single        chrominance channel 1192 can be used and filtered for color        using rotating color filters or an electro-optic filter before        being sent to a separate intensifier 816.    -   The output of the intensifiers 816 can be overlaid and displayed        760 on a CRT or other output device, or optically combined for        direct viewing.

In contrast to systems which use temporal or spatial scanning of theentire bandwidth and a single intensifier, the illustrative color nightvision system 700 d seen in FIG. 16 has the advantage of high spatialand temporal resolution of the luminance channel, which is the mostimportant channel for vision.

Further improvement can be achieved by treating the chrominance 1192 asluminance 1190 in very low light levels, e.g., reverting to aconventional monochromatic display.

FIG. 17 shows an illustrative schematic diagram of an embodiment of acolor night vision system 700 e that includes two intensifier tubeassemblies 810. The illustrative color night vision system 700 e seen inFIG. 17 reduces the number of required intensifiers 810 to a total oftwo (one per eye). The first intensifier 810 contains filters 812distributed throughout its area for cyan, magenta, yellow, and infrared(CMY+IR), interspersed with a large majority of conventional greenfilters 812. The intensifier output is directed to a 45-degreebeamsplitter 1222. One resulting beam reflects to a camera 1224, whichsends the CMY+IR channels to a computer 1228.

The computer reconstructs red and blue information from the CMY+IRfilters and outputs to an OLED display 760, e.g., 760 a. The other beam1126 continues to a second beamsplitter, where the OLED display 760 a iscombined with the directly viewed green channel information for viewing.

The second intensifier 810 is coupled to a neutral density filter 812,which reduces its light output to match the loss suffered by the firstintensifier 810 from the first beamsplitter 1222. A beamsplitter 1230allows combination of its direct view green channel with red and blueinformation from a second OLED 760, e.g., 760 b, produced by the samecamera 1224 and computer 1228 output as the first intensifier 810.

The OLED input can also be used to place textual or graphical overlayson the color night vision camera's output. Alternatively, direct viewingcould be eliminated by placing a camera 760 on the second intensifier810, and presenting the combined channel output on the OLED 760 on bothsides.

Yet another embodiment removes the CMY+IR filters from the eyepieceintensifiers 810, instead adding a third intensifier 810 with a denseCMY+IR pattern. The output from that intensifier 810 feeds the OLEDs 760in each eyepiece intensifier 810.

FIG. 18 shows an illustrative schematic diagram 1260 of an alternateembodiment of a color night vision system 700 f that includes twointensifier tube assemblies 810, which characterizes the color input byeach fiber of an intensifier, enabling the system 700 f to reproduce acolor image 1268 using only one intensifier 810 per eye. A Bayer pattern1262 preserves the appropriate proportion of green to red and blue.

A Bayer pattern 1262 is placed on the input (front) end of each imageintensifier 810, which filters each pixel's color to red, green or blue.A corresponding Bayer pattern 1265 on the output end of the intensifier810 can be used to reconstitute the color pattern. When combined with awhite phosphor 1264, the color information is re-encoded 1266, resultingin a color image 1268.

There are several ways to accomplish this:

In one embodiment, light from the phosphor 1264 of the device 10 is usedto expose a photo resist material. This allows the generation of coloreddyes exactly matching the required positions of the individual dyecells, accommodating any distortions induced by the intensifier. Simplyilluminating the capturing Bayer pattern 1264 with red, green and bluelight at different steps of the process allows a staged manufacturingprocess.

Alternately, a machine vision camera and projector assembly can be usedto observe the patterns on the phosphor 1264 of the intensifier 810 thatthe Bayer pattern 1262 created. Light is then projected of theappropriate shape and wavelength to accommodate the photolithographyprocess.

This information can also be used to characterize the color of eachpixel input to a CCD imager, e.g., 820, which accepts the output fromthe intensifier 810. The imager 820 can then re-encode the colordigitally.

Another method is to do away with the Bayer filter 1262 at the inputend, and simply send the output image to a fiber optic bundle, which isthen randomly divided into N bundles, one per color required. Eachseparated bundle's fibers are taken from throughout the spatial extentof the entire bundle.

The opposite, separated ends of the bundles are each imaged by CCD chipsor other devices, through a color filter. For example, the fiber can bedivided into three bundles, which send light through red, green and bluefilters 812, for an RGB color scheme.

Since each bundle is a random and equally distributed selection offibers, the color night vision device 10 must be trained as to whichfibers correspond to which colors. For example, the device 10 can bepointed at a pure green background, and the spatial location of eachsubsequent green pixel recorded in the device's permanent memory. Thisprocess can then be repeated for the red and blue bundles.

This arrangement decreases fabrication costs in favor of computationaland fiber costs, which are relatively inexpensive.

Yet another method addresses the fact that each filter cuts out either ⅔of the available photons (RGB scheme) or ⅓ (CMY scheme). Furthermore,the vast majority of image detail lies in the luminance channel, withthe chrominance channel adding very little information. Such anembodiment implies that most color information can be discarded, infavor of higher collection of photons for greater sensitivity, whilemaintaining sufficient color information for a reasonable image.

Therefore, to retain sufficient color information while retaining asmuch sensitivity as possible, a glass disk can be placed in the imagepathway, with only a small minority of its area comprised of widelydispersed RGB or CMY color filters, while the remainder is comprised ofclear glass to pass all photons. The color filters furnish sufficientcolor information to reconstruct the approximate locations of color inthe scene, at the expense of less well-defined boundaries betweencolors.

This sparse color mask is duplicated on a second glass plate with anidentical but mirror image of the pattern, such that the two plates canbe aligned. The color resolution can be doubled at any time by rotatingone of the filters such that the color elements are misaligned;therefore twice as many color filters are in the image path with acorresponding reduction in photon capture.

Systems and Methods for Noise Reduction

Noise reduction in static amplified images is a well-defined art, drivenin part by the need for sensitive astronomical instruments. Sincethermal and other noise is always present, one technique is to subtracta dark image (where the shutter is closed) from the normal image, thusremoving much of the noise.

FIG. 19 is a process flowchart of an illustrative method 1400 forreducing image noise in a color night vision device 10, by comparingoutput of the color channels. An illustrative color night vision device10 is constructed with a minimum of three image amplification channels710 for monocular vision, or a second set of three channels 710 forstereo vision. Cyan, magenta, and yellow filters 812 are placed over thethree channels 710. CMY is used, because, unlike RGB, the color curvesoverlap such that each point along the spectrum appears in at least twochannels. For example, any color in the red spectrum appears in both themagenta and yellow channels and blue is covered by cyan and magenta.

The output 1402 of each channel 710 is converted 1404 to a digitalsignal. A processor compares 1406 the content of each channel 710. Sincethe CMY curves completely overlap, any actual object, even on a pixellevel, must appear in two channels 710. If changes occur in only onechannel 710, that is an indication of noise. These speckles are thenfiltered out 1408, and the output is converted 1410 to RGB space, fordisplay 1412 to the viewer V. Although the illustrative process seen inFIG. 19 removes the bright, white-appearing speckling, somelower-intensity colored speckling still appears.

Note that this process 1400 can apply not only to color night visiondevices 10, but can be implemented with any solid-state imaging device.

In some embodiments, a second set of detectors can be used to gather1422 additional light in the form of RGB components. For example, cyanlight (green and blue) is transmitted to the cyan channel 710 above, byplacing a cyan dichroic filter 812 in front of an image intensifier 816,which reflects or absorbs any non-cyan light, i.e., red. Normally, thisreflected light is discarded, but any additional light is welcome in alowlight situation.

If the filter 812 is placed at an angle relative to the intensifier 816,such as at an angle of 45 degrees, the reflected light can betransmitted to an additional intensifier 816. Analogous filters areplaced in front of intensifiers 816, to separate magenta and yellow, andin this way, red, green, and blue light components are gathered 1422 inthe three additional intensifiers 816. These RGB components can then becombined 1424 with the CMY components for viewing 1412, either opticallyor on a display 760.

FIG. 20 is a process flowchart of an illustrative method 1400 forreducing image noise in a color night vision device 10, using aplurality of buffers. In the illustrative method seen in FIG. 20, animage 702 is captured 1702 at a given frame rate, for example 30 framesper second, by an image amplification device and digital camera. Theprocessor contains two frame buffers, “current” and “accumulation”. Eachframe is collected 1504 first in the current buffer, then sent 1506 onthe next cycle (for example 1/30 second) to the accumulation buffer. Thelatter buffer merges 1508 the pixel-by-pixel information of allcollected frames into a composite image. The resulting average reducesnoise, but any moving object blurs or disappears.

The image can then be displayed 1510, wherein the image is a weightedcomposite of the two buffered images. The composite image is a tradeoff;as the accumulated buffer is given more weight, noise is reduced, butmoving objects disappear; as the current buffer is given more weight,moving objects are seen but noise increases.

To help alleviate this tradeoff, a spatially sensitive dynamic filtercan be used to weight static and moving portions of the imagedifferently. For example, a spinning object can occupy the upper rightportion of the frame. Most of the image is static, and the accumulatedbuffer is highly weighted. However, the upper right portion around thespinning object is weighted towards the current buffer. Thus, the staticpart of the image retains low noise, while the moving upper rightportion can still be viewed, at the cost of increased noise in thatportion only.

Such a filter can be implemented by the following steps: First, eachcurrent image is blurred, to remove high frequency information,including noise (which is pixel-sized). This is compared to a blurredimage of the image in the accumulated buffer. Any difference can beinterpreted as a moving object. The area around the difference is thenweighted towards the current buffer. Thus, the static part of the imageretains low noise, while the moving upper right portion can still beviewed, at the cost of increased noise in that portion only. These areascan be distinguished by masking filters that separate the moving andnon-moving portions of the image.

Additionally, the masking filters can be further weighted by otherparameters. For example, the sensitivity of the masks can vary, based onwhether they are located towards the bottom or top of the image. Movingobjects towards the bottom of the field of view can be assumed to belarger, while those on the top are typically smaller. Thus, a moresensitive mask may be needed to detect smaller moving areas towards thetop of the field of view.

FIG. 21 is a process flowchart of an illustrative method 1600 forreducing image noise in a color night vision device 10, which applies1604 a non-local image denoising algorithm to reduce the noise. In someembodiments, the applied 1604 algorithm is described by A. Buades etal., “A non-local algorithm for image denoising”, Proc. IEEE CVPR, vol.2, pp. 60-65, 2005. In some embodiments, the noise reduction isperformed using an “accelerated” variation of the algorithm described byWang, Jin et al., “Fast non-local algorithm for image denoising” ImageProcessing, 2006 IEEE International Conference on, pp. 1429-1432. Insome embodiments, the algorithm is extended or enhanced to capitalize onthe CMY structure of the incoming 1602 image data.

Spatial or Temporal Integration for Improved Color Night Vision

The color night vision system 700 increases the sensitivity of a colornight vision system by integrating one or more color channels 710 in thespatial and or temporal dimensions. This increases the overallbrightness of the available image, at the possible expense of reducedresolution due to motion artifacts or spatial limitations of the imagingsystem.

By comparing the images in different color channels 710, the color nightvision system 700 preferably modulates the amount of color informationavailable after spatial and/or temporal integration has taken place. Forexample, the blue channel 710 b can be captured at a relatively lowframe rate (and a correspondingly longer integration time). Theresultant composite image can therefore have more blur in the bluechannel 710 b. In some embodiments, this blue signal can be added as abrightness value to all channels, with a resultant loss of saturation.As the data in the blue image becomes richer with information, it cancontribute more color data.

CMY-Filtered Field Sequential Color Night Vision for Improved Brightness

An existing type of color night vision device rotates a disk of red,green, and blue color filters in front of a standard night vision scope.The difference in intensity produced by differently colored objects isinterpreted as color by the user.

Alternatively, a corresponding, matching disk of rapidly rotating outputfilters can output color fields in sequence, with the combined image isa color representation.

FIG. 24 is a simplified schematic diagram 1960 of an illustrativeembodiment of a system for CMY-Filtered sequential color night vision.FIG. 25 is a detailed view 1980 of light transmitted through an inclinedfilter 1964. FIG. 26 is a flowchart of an illustrative method 2000 forCMY-filtered sequential color night vision.

In essence, this method 2000 scans colors over time. However, each colorfilter 1964 dramatically reduces the number of photons captured, e.g.,brightness, since photons not of that color are excluded. Further, RGBfilters have a narrow pass range, further cutting back on photonscaptured.

To improve light gathering for such a rotating filter type of device,Cyan-Magenta-Yellow (CMY) filters 1964, e.g., 1964 c, 1964 m, 1964 y,are fitted to a rotating disk 1962. CMY filters each cover about ⅔ ofthe spectrum, vs. ⅓ for conventional RGB filters. This effectivelydoubles the number of photons passed through each filter.

To capture even more photons, each filter 1964 is tilted 1982 at 45degrees with respect to the incoming light 1968, with its interferencelayers being suitably compressed to continue the pass the appropriatefrequencies. While light of the appropriate frequency range 1974continues to be passed through the filter as above, the rejected photons1970 (R for C, G for M, and B for Y) are reflected from the filter, andare captured by a sensor. The rejected photons are then recombined withthe passed light, using a similar arrangement on the viewing side. Thisfurnishes the maximum possible brightness and sensitivity in anon-direct view embodiment.

Motion Compensation for Temporal Scanning Color Night Vision

Another issue with a conventional rotating/temporal scanning filter typeof night vision device, as discussed above, is that if the device is notstationary, relative to the target, such as a device used on a vehicleor aircraft, color smearing can occur, as each filtered image is viewedin a slightly different location than the last.

To counter this, some embodiments of the color night vision device 10can be fitted with rotating optical filters, or electronic temporalscanning, to generate color images. FIG. 27 is a simplified schematicdiagram 2040 of motion compensation for temporal scanning color nightvision. FIG. 28 is a flowchart of an illustrative method 2080 for motioncompensation for temporal scanning color night vision.

In some embodiments 10, the mechanism for encoding the information canbe a motorized filter wheel, or a switched liquid crystal device. Byviewing a white phosphor viewing screen on the rear of the color nightvision device 10, through a similar mechanism, this color information ispresented, such as directly to the human eye, or to spatially encodedcolor sensors.

Using the known characteristics of the optics of the color night visiondevice 10, a rotational sensor and accelerometers on the camera 10 cantrack the camera's motion 2049, while a compensating algorithm can shifteach of the resulting three separately colored images to maintainalignment. However, this technique cannot compensate for situationswhere the viewed object itself is moving.

Circular Filter Variable Between Color Purity and Brightness

Color night vision systems are typically not adjustable to trade offcolor information versus sensitivity. This can be a useful feature, suchas in especially low-light situations, where color is less importantthan gathering the maximum number of pixels.

FIG. 22 is a schematic view 1900 of a circular filter disk 1902 for acolor night vision camera 10, wherein the filter disk 1902 is in a firstposition 1910 a, and can be rotated 1942 (FIG. 23) in an image path1904, to vary the effectiveness of the color filter 1908. FIG. 23 is aschematic view 1940 a circular filter disk 1902 for a color night visioncamera 10, wherein the filter disk 1902 is in a second position 1910 b,and has been rotated 1942 to a second position 1910 b, to vary theeffectiveness of the color filter 1908.

The circular filter region 1906 sees only a portion of the disc 1902, tovary the effectiveness of the color filter 1902. For example, the disk1902 can be constructed with a cyan filter 1908 that, in a firstposition 1910 a, is 100% effective in a particular radial direction,rejecting all red light. As the disk 1902 is rotated 1942 about itscenter, the cyan filter 1908 passes more red light, until a point atwhich no photons are rejected. Thus the effectiveness of the filter 1902can be infinitely variable, between color purity and maximum brightness.

Complementary filters are used in the image path for other colors,either CMY or RGB. Alternatively, the disc can be used to vary blockageof infrared light.

In practice, the effect can be achieved by producing the filter in acheckerboard pattern with blocks alternating between filter blocks andclear pass blocks, e.g., a variable density dichroic color separator. Atthe first position, the filter blocks fill the entire area presented tothe image path; as the disc is rotated, the filter blocks become smallerwhile the clear pass blocks become larger, until the entire area in theimage path is clear.

Note that any and all of the embodiments described above can be combinedwith each other, except to the extent that it may be stated otherwiseabove or to the extent that any such embodiments might be mutuallyexclusive in function and/or structure.

Although the present invention has been described with reference tospecific exemplary embodiments, it will be recognized that the inventionis not limited to the embodiments described, but can be practiced withmodification and alteration within the spirit and scope of the appendedclaims. Accordingly, the specification and drawings are to be regardedin an illustrative sense rather than a restrictive sense.

What is claimed is:
 1. An imaging device, comprising: a filter element,wherein the filter element receives and filters an incoming imagesignal; one or more image intensifiers for receiving the filteredincoming image signal, wherein each of the one or more imageintensifiers has an analog output associated therewith; one or moreaccelerometers, wherein the one or more accelerometers track motion ofthe imaging device; a mechanism for converting the analog outputs of theone or more image intensifiers into corresponding digital image signalsfor processing; and an image processor programmed to utilizeaccelerometer output to shift the digital image signals and provide theshifted digital image signals for viewing or further processing.
 2. Theimaging device of claim 1, wherein the filter element is operablycoupled to a motor and is rotatable with respect to the one or moreimage intensifiers.
 3. The imaging device of claim 2, wherein the filterelement further comprises a housing which houses at least one colorcomponent filter.
 4. The imaging device of claim 2, wherein the filterelement further comprises a housing which houses at least three colorcomponent filters, and each of the color component filters has an angledconfiguration with respect to the filter element housing.
 5. The imagingdevice of claim 3, wherein one or more of the color component filtershas an angled configuration with respect to the filter element housing.6. The imaging device of claim 4, further comprising a sensor arrangedto receive reflected light from each of the angled color componentfilters.
 7. The imaging device of claim 4, wherein each of the colorcomponent filters are CMY filters respectively.
 8. The imaging device ofclaim 4, wherein the color component filters are RGB filtersrespectively.
 9. The imaging device of claim 5, further comprising asensor arranged to receive reflected light from the angled colorcomponent filters.
 10. The imaging device of claim 6, further comprisingat least one outputting mechanism configured to be operable to provide acombined channel as an output of the imaging apparatus, wherein thecombined channel comprises the shifted digital image signals and thesensor received reflected light.
 11. An optical processing methodcomprising the steps of: filtering an incoming image signal via a filterelement, wherein the filter element receives an incoming image signal ofan environment; intensifying, via one or more image intensifiers, thefiltered incoming image signal and producing an analog intensified imagesignal output from each of the one or more image intensifiers; sensingmotion of the imaging device, via one or more accelerometers; convertingthe analog intensified image signal outputs of the one or more imageintensifiers into corresponding digital image signals; and shifting thedigital image signals based on the motion sensed by the one or moreaccelerometers; and providing the shifted digital image signals forviewing or further processing.
 12. The optical processing method ofclaim 11, further comprising the step of: rotating the filter elementwith respect to the one or more image intensifiers, wherein the filterelement has a housing that is rotated by actuation of a motor.
 13. Theoptical processing method of claim 12, wherein the filter elementhousing houses at least three color component filters, and each of thecolor component filters has an angled configuration with respect to thefilter element housing.
 14. The optical processing method of claim 13,further comprising the steps of: reflecting a first light component withan angled first color component filter of the at least three colorcomponent filters; reflecting a second light component with an angledsecond color component filter of the at least three color componentfilters; and reflecting a third light component with an angled thirdcolor component filter of the at least three color component filters.15. The optical processing method of claim 14, further comprising thestep of: detecting, via a light sensor, reflected light components fromeach of the angled color component filters.
 16. The optical processingmethod of claim 15, further comprising the steps of: combining thedetected light components with the shifted digital image signals, andproviding the combination as output for viewing.
 17. The opticalprocessing method of claim 16, wherein the first light component isnon-cyan light, the second light component is non-magenta light, and thethird light component is non-yellow light.
 18. The optical processingmethod of claim 16, wherein the first light component is non-red light,the second light component is non-green light, and the third componentis non-blue light.
 19. The optical processing method of claim 15,further comprising the step of: shifting the detected light componentsbased on the motion sensed by the one or more accelerometers.
 20. Theoptical processing method of claim 19, further comprising the steps of:combining the shifted light components with the shifted digital imagesignals; and providing the combination as output for viewing.